Transience Bounds for Long Walks

Let G = (V; E) be a directed graph with n vertices. For each e 2 E there is an associated reward r e ; the n n matrix A = a ij ] is deened by a ij = r ij if ij 2 E and a ij = ?1 if ij 6 2 E. The max-algebra system of equations y(k) = y(k?1) A is a deterministic dynamic programming recursion for the maximum total reward y i (k) when the system is in state i at stage k and the one-stage transition reward matrix is A; when r ij represents the duration of activities that begin and end with the occurrence of events i and j, respectively, this is a discrete-event simulation. Max-algebra systems have also been used to simulate certain automated manufacturing systems. If G is strongly connected, the solution exhibits periodic behavior after an initial transient: if is the maximum cycle mean in G, then y(k+d A) = y(k)+d A 1 for all k K A , where K A and d A are the max-algebra transient and period of the matrix A. For given initial conditions y(0), the transient K and period of the system may satisfy K < K A and < d A. We show that for all k > n 2 , there exists a maximum reward s; t-walk in G of length k which consists of an s; t-walk of length at most n 2 and copies of a cycle with maximal mean reward. We use this result to (1) derive a bound on K A that is tight to within a constant factor, and give polynomial algorithms for computing K and and (2) develop a strongly polynomial algorithm for solving deterministic dynamic programming problems. We give an analogous result for the structure of walks of maximum-discounted reward, and use this result to develop strongly polynomial algorithms for solving nite and innnite horizon-discounted deterministic dynamic programming problems.